Repository navigation
fix!: table names stay readable in logs, Unity Catalog tag values as written, plan refuses a blocked residency - #868
Merged
Conversation
…able name fluid apply printed "databricks.load table=***REDACTED*** rows=340" for northwind_towers.bronze_smart_meters.smart_meters and the real name for a table whose last part had 11 characters. The prefix-agnostic _JWT_RE masked any three dotted runs of 12 or more characters, and a catalog.schema.table name (or project.dataset.table, DB.SCHEMA.TABLE) has that shape. Such a run is now masked only when its first part is a JOSE header: base64url of a JSON object, the check detect-secrets' JwtTokenDetector.is_formally_valid makes (Yelp/detect-secrets plugins/jwt.py). The eyJ-anchored rule, which gitleaks' jwt rule and trufflehog's JWT detector also anchor on, now covers a JWE (a direct-encryption one has an empty encrypted key) and short or empty payloads and signatures. One redact_jose_tokens() serves both redaction layers: the Snowflake twin calls it, as it imports _URL_USERINFO_RE, so they cannot drift. The rules start where no base64url character precedes them instead of at \b, which matched after every '-' and made the scan quadratic on a-a-a-... input (200k characters: about 30 s before, 0.07 s now). keep_dotted_names goes: logs and printed errors apply the same rule now.
…s them The governance labels became Unity Catalog tags with GCP's label slugs: lowercased, spaces and punctuation as '-', cut at 63 characters (business_value = 'a-single-unrenewed-ground-lease-can-strand-a-multi-tenant- tower', fluid_jurisdiction = 'us'). A UC tag value takes up to 256 characters with case, spaces and punctuation, and may not start or end with a space (docs.databricks.com, "Apply tags to Unity Catalog securable objects"). governance_labels() gains verbatim_values: the Databricks module asks for the contract's text, stripped, with the regulatory frameworks joined by ", ", and refuses a value over 256 as databricks-tag-value-too-long (one _too_long refusal now serves the product tags too). Keys keep GCP's label alphabet, each '-' an '_'; GCP labels and AWS tags are unchanged. An existing deployment updates in place: databricks_entity_tag_assignment's tag_value carries no RequiresReplace at provider v1.140.0 and Update sends UpdateMask "tag_value" (entity_tag_assignment/resource_entity_tag_assignment.go), and the address and import id depend on the key alone. Pinned by a fake- workspace e2e (slug values, then the real ones: +0 ~2 -0, nothing replaced) and observed on Free Edition: +0 ~17 -0, the table's rows untouched, verify green on all 27 tags.
…default
fluid validate and fluid plan --check-sovereignty refused a contract whose
binding region its strict sovereignty denies ("Region 'us-west-2' is
explicitly denied by sovereignty policy"), but plain fluid plan wrote a plan
for it, and --check-sovereignty itself wrote the plan before it exited 1.
plan now runs the built-in policy engine, the checker validate runs, before
it plans, and refuses (exit 1, sovereignty_violation, no plan.json) whenever
validate would: strict findings, or a region in deniedRegions, which blocks in
every mode. It builds no provider, so a plain plan stays off the cloud SDKs
(Command Center's plan endpoint relies on that) and prints nothing when
nothing blocks. --check-sovereignty keeps asking the provider's hook and
printing the verdict, now before the write. --no-check-sovereignty
(argparse.BooleanOptionalAction) skips the check and logs a WARNING with the
event sovereignty_check_skipped, the shape of apply's --no-verify-plan-binding;
it changes nothing in apply, which still refuses the placement on Databricks.
The design follows HCP Terraform's policy enforcement levels: a mandatory
policy stops the run, and an override is explicit and recorded.
BREAKING CHANGE: fluid plan exits 1 and writes no plan for a contract its
sovereignty policy blocks; pass --no-check-sovereignty to plan it anyway.
…nty never weaker From the branch's own review. - _JWT_TWO_SEGMENT_RE still started at \b, so eyJ-eyJ-... stayed quadratic (200k characters: 9.75 s through redact_secret_text, 0.30 s now), and the linear test fed eyJ_ (no \b after '_'), so it could not see it. The rule now starts where _JWT_ANCHORED_RE does; _JWT_RE takes two runs as well as three, so a two-segment token glued after '-' is still masked by its header. The test feeds eyJ- too. - _JWT_ANCHORED_RE took up to two more segments greedily, so "got <jwt>.Next" lost "Next". It now takes a JWS (three segments) or a JWE (five). - --check-sovereignty asked the provider's hook first, so a hook that reported nothing passed a contract the new default refuses. It now runs the default gate first. --no-check-sovereignty also says so on the console. - The changelog no longer says a failing plan stage leaves no plan file: one an earlier run left at --out stays.
- fluid plan's sovereignty refusal printed its findings through Rich markup, which took each finding's "[expose_id]" for a tag and dropped it; findings print literally now (here and in --check-sovereignty's report). The refusal also carries the findings as the CLIError's context errors and message, the shape local_plan_validation_failed has: Command Center's plan endpoint surfaces an engine refusal that has them, and fell back to an ungated parse-only plan for one that had not. - _JWT_RE takes up to five segments, empty ones included, so a JWE with direct encryption whose header is written with a space or a newline is masked whole again (origin/main masked its last three segments; the branch left it). Its header check still decides, so names stay readable. - governance_labels' verbatim values spell a YAML true/false as written and make no tag for null: fluid generate iac emits without the schema gate, which allows only strings, and wrote Python's "True".
From the security judge's re-check. A token glued after a dotted name
(northwind_towers.x-<header>.<payload>) and a header written with spaces or
newlines glued to a word (token_eyAi..., x-ewog...) were masked on origin/main,
whose rule masked any three 12+ runs, but not on the branch, which looked for
a header only in the run's first segment and only at "eyJ".
_mask_if_jose now checks every segment of the run and masks from the first
that holds a header, keeping the name before it. A glued header starts at an
'e' after '-' or '_' or at an "eyJ" (base64url of '{' always starts with 'e'),
six tries at most per segment. _JWT_RE takes the whole dotted run, and
_is_jose_header requires the "alg" member RFC 7515 and RFC 7516 (4.1.1) make
mandatory, so a short JSON part of a name such as e30 ({}) is no header.
200,000-character adversarial runs stay under 0.5 s.
… default #861's clean() asked redact_secret_text for keep_dotted_names=True, the keyword this branch removes: every log and printed error now keeps a dotted name and masks a JWT by its header, so the call passes nothing extra. Without this, every Databricks job-run message raised TypeError (39 tests in tests/build_runners/test_databricks_job_run.py after rebasing onto #861).
…ion-tags-residency
github-actions
Bot
added
provider
📄 Documentation ReminderThis PR appears to be missing a documentation reference. Our docs live in a separate repo. Please update the PR description with one of:
See the Contributing Guide for details. |
![github-code-quality[bot]](https://avatars.githubusercontent.com/u/9919?s=60&v=4){kind=small}
github-actions
Bot
removed
the
needs-docs
…nation in the catalog From the code-quality review of #868. The two sovereignty_violation suggestions split over adjacent literals now use the file's own parenthesised "+" form, and the tests import fluid_build.observability.secret_redactor, the Snowflake logging module and fluid_build.cli.plan one way each. No behaviour change; the baseline moves line numbers only.
…o fix/databricks-redaction-tags-residency # Conflicts: # .secrets.baseline # CHANGELOG.md
Test (Python 3.14) failed test_the_words_after_a_token_are_kept with "got ***REDACTED******REDACTED***.Next step". The exact-match registry is process-wide, and a masking test in the same xdist worker had registered the base64url of bytes(range(32)), which is this test's signature, so layer 1 masked it first and the JOSE rule then masked "header.payload." on its own. The token was masked whole either way; the exact-string expectation was not. Both JOSE test classes now set the registry aside for each test and restore it after (forget_known_secrets / register_secrets). Reproduced with that key registered first: the old file fails 1, the new one passes 55.
…o fix/databricks-redaction-tags-residency # Conflicts: # .secrets.baseline # CHANGELOG.md
fas89
added a commit
that referenced
this pull request
Nov 13, 2026
This was referenced Nov 13, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Three defects seen live on 12 Nov 2026 (data-product-forge 0.29.0, Databricks Free Edition), fixed in one PR because each stayed small. The PR has three feature commits, four from the review rounds, and one that keeps #861 working (it called the removed
keep_dotted_names); it squash-merges to one.What changed and why
1. Log redaction stopped masking fully qualified table names.
fluid applyprinteddatabricks.load table=***REDACTED*** rows=340fornorthwind_towers.bronze_smart_meters.smart_meters, but printed the real name of the next table, whose last part had 11 characters. The cause was a prefix-agnostic JWT rule (_JWT_RE) that masked any three dotted runs of 12 or more characters, and acatalog.schema.tablename has exactly that shape.Now such a run is masked only when its first segment base64url-decodes to a JSON object, which is how a JOSE header looks. This is the check detect-secrets'
JwtTokenDetectormakes. TheeyJ-anchored rules, which gitleaks and trufflehog also anchor on, still run first, and they now cover:token_eyJ…).The global redactor and its Snowflake twin share one
redact_jose_tokens(), so the two cannot drift. The rules now start where no base64url character precedes the match, instead of at\b. That ends the quadratic scans ona-a-a-…andeyJ-eyJ-…: 200,000 characters took 30 to 90 s before and now take under a second.keep_dotted_namesis removed, since logs and printed errors now apply the same rule.2. Unity Catalog tag values are written as the contract states them. Before, a label value reached Unity Catalog as GCP's label slug: lowercased, with spaces and punctuation replaced by
-, cut at 63 characters (business_value = 'a-single-unrenewed-ground-lease-can-strand-a-multi-tenant-tower',fluid_jurisdiction = 'us').governance_labels(verbatim_values=True)keeps the contract's text, minus surrounding whitespace, which Unity Catalog refuses. Frameworks are joined asGDPR, CCPA.databricks-tag-value-too-long, never cut.-still becomes_. GCP labels and AWS tags are unchanged.What this means for an existing deployment:
tag_valuehas no RequiresReplace at provider v1.140.0, and the provider's Update sendsUpdateMask: "tag_value". The address and import id depend on the key alone. So the next apply updates each tag in place (~) and destroys nothing. The CHANGELOG says this.3.
fluid planrefuses a contract its own sovereignty policy blocks, by default. Founder's choice: gate plus skip flag.fluid validateandplan --check-sovereigntyrefusedRegion 'us-west-2' is explicitly denied by sovereignty policy, yet plainfluid planwrote a plan for it. Now plan runs the policy engine, the same checkvalidateruns, before it plans:ERR_SOVEREIGNTY_VIOLATION) and writes no plan.errorsandmessage. That is the shapelocal_plan_validation_failedhas, so Command Center's plan endpoint surfaces the refusal. Without it, CC fell back to an ungated parse-only plan.--check-sovereigntyruns the same check and then asks the provider's hook. A hook that reports nothing can no longer pass whatvalidaterefuses.--no-check-sovereignty(BooleanOptionalAction) writes the plan anyway, says so on the console and logs a WARNING (sovereignty_check_skipped).Apply is unchanged. On Databricks and GCP it already refuses such a placement at emit; verified on the real overlay.
Borrow before build
_JWT_ANCHORED_REalready existed (console only)._product_tagsalready wrote verbatim values and refused over 256.--no-verify-plan-bindingand--allow-data-lossare the escape-hatch precedent. Command Center origin/main has nokeep_dotted_names, and its label mirror covers AWS and GCP only.resource_entity_tag_assignment.go:tag_valueis updated in place.jwtrule (ey…), detect-secretsJwtTokenDetector(eyJplus JSON decode), trufflehogjwt.go(eyJ|ewogIC|ewoid).--soft-fail.Validation
ruff, the dead-code gate,black==24.10.0, product guardrail, license headers,lint-imports(6 kept),detect-secrets==1.5.0on changed files, strictmypyon the hotspots (secret_redactor.pyincluded), and the agent-skill reference--check. All clean.--randomly-seed=20261112,-n 6): 35,101 passed on the final rebased head. Every failure is the documented shared-venv baseline (lazy core commands ×2, dotenv dlt ×4, duckdb sandbox workspace root ×2, the MCP Snowflake-login flake, cycles budget e11 under load, and the dbt-prefix PATH test, which fails on pristine origin/main too) plus 2 bashlex collection errors.tests/fragments/test_compat_gate.py, run alone as the shared venv needs: 45 passed.+0 ~2 -0, nothing replaced.--env region-onlyplan: exit 1, no plan written.--env databricks(declared override): exit 0.--no-check-sovereignty: exit 0, with a warning.applyof the skipped plan:SovereigntyViolationError.business_value,grainandstandardsverbatim,fluid_jurisdiction = 'US'.fluid_iactest_small, created and then dropped with CASCADE;workspaceandnorthwind_towersuntouched:databricks.load table=fluid_iactest_small.bronze_smart_meters.smart_meters rows=3.+0 ~17 -0under this code, then applied.information_schema.table_tagsshows the values verbatim: the 256-character boundary,;,,,:,/,=,( ),Zürich. The rows were untouched.fluid verifypassed on all 27 tags, and a second dry-run gave+0 ~0 -0.ripwire --quality-delta: the real regressions are fixed (governance_labelscomplexity 30→31,_uc_tagslength, test duplication). It still exits 2 on dead-code name-resolution artifacts in untouched files (was=0, now=0), churn and 27–37-token coincidental clones.Reviews
/security-review: no HIGH or MEDIUM. The Opus judge on the review-round diff: no HIGH/MEDIUM in the fix diff or the whole branch, after probing about 37 credential shapes against origin/main in both layers. Its two below-threshold residuals (a token glued after a dotted name, a glued header written with spaces) are fixed in 9496aac and re-checked: no issue.
/world-class-inspection (Diff): CONCERNS, with one P2:
eyJ-eyJ-…was still quadratic through the two-segment rule, and the test missed it. That P2 and the P3s are fixed in 76e9c153 and re-verified./defect-scan (Diff): no Critical, High or Medium in the code this change touches. Three LOW issues are fixed in 776bf09f:
generate iac;The Command Center fallback is closed from forge's side.
Re-check round: one auditor confirmed all 9 fixed findings with 316 tests and a 150,000-name fuzz that masked no name; every entry point stays under 1 s on 200,000-character input; no new Critical/High/Medium.
Integration with the PRs that merged meanwhile
redact_secret_text(..., keep_dotted_names=True). With the keyword gone, that raised TypeError in 39 of its tests. 8b375e6 drops the keyword: keeping dotted names is now the default, which is what that call asked for.errorsit already reads.Documentation
docs/cli/plan.md: plan checks sovereignty by default and refuses whatfluid validaterefuses (ERR_SOVEREIGNTY_VIOLATION, no plan written);--check-sovereigntyadds the provider hook;--no-check-sovereigntyskips with a warning.databricks-tag-value-too-long); an existing deployment's tags update in place on the next apply.upgrading.md: both behaviour changes, under the not-yet-released section.Follow-ups (pre-existing, not in this PR)
Foo Barandfoo-bar) silently overwrite each other ingovernance_labels. This affects GCP, AWS and Databricks and is identical on main.listen(5)backlog overflows under tofu's 10-way refresh on macOS, which sends an RST, so local fake e2e runs sometimes fail withwrite: broken pipe. Linux CI drops the SYNs and retries instead.